Today's software products are increasingly complex, often capable of performing numerous tasks simultaneously and operating upon multiple platforms and operating systems. In addition, many software products in use today are graphics intensive and employ graphical user interfaces (GUIs) that correlate graphics elements (e.g., buttons, dialog boxes, menus) to the executable features of the products. This allows a user to easily navigate through an application to perform a desired task. Invariably, to ensure the reliability of software products, comprehensive testing and debugging are key components to the software development process. In general, software testing and debugging are the processes of identifying and correcting programmatic or operational defects within a software product.
Many defects that occur within a software product are not fully exposed until the application is actually in execution, referred to as being “in a runtime state.” For example, a newly developed application may stall or produce an error message when its user selects a certain button or menu from a GUI during runtime. Other common errors that occur within GUI-based applications are visual defects, such as text truncation and spelling errors. Such errors occur even more frequently in software products that have been “localized.” Software is localized when it has been translated or adapted from its original language to a new language to accommodate new users. Oftentimes, in adapting the software to the new language, one or more text strings are left untranslated. To identify these errors prior to deployment into the intended market, the functions and features of the product should be tested exhaustibly during runtime.
One method of testing the features of an application is to employ an automated test application, commonly referred to as a “monkey”. Monkeys are executable applications that contain instructions for automatically exercising, or invoking, the features of an application under test. Monkeys allow executable states, or actions, of the application to be exposed during runtime and subsequently verified for proper operation by the tester. For example, a monkey can be used to automatically activate menus, buttons, dialog boxes, and other graphics elements that represent the features of a typical GUI-based application as it executes. In operation, the monkey randomly selects an action or graphics element from within the GUI—perhaps the “File” button—resulting in a drop down menu of other user options (e.g., New, Open, Exit) being displayed to the screen. This random selection of actions or graphics elements is continued until the software tester terminates the process manually or until the monkey encounters an un-recoverable system or programming error (e.g., invalid command, system crash).
The monkey is a “dumb” test, meaning it has no model or understanding of the software under test. It simply activates any executable action or graphics element within the software package without regard to the operation underlying the action or graphics element. This characteristic makes monkeys cheap to design and easy to program, as they need not be customized to fit the application under test. Also, the relative ease of designing test monkeys makes them particularly useful during the early stages of testing, where most defects or errors are found. Furthermore, by automating the GUI-testing process, the software tester does not have to manually invoke each of the actions and graphics elements that make up the application. This reduces the time required for testing user interface features and eliminates any human error that could result from manual execution. All of these features are particularly useful in the testing of graphics-oriented software packages having several executable features and options.
Monkeys, however, are not without some inherent limitations. Because conventional monkeys and other automated testing techniques perform random acts upon the software, unnecessary test repetition can occur. This results in significant time being consumed in order to fully exercise an application under test. Even further, repetition compounds the cost of testing a product, especially those products having several features and associated graphics elements that are accessible from a GUI. Typically, monkey tests must run for several days at a time in order to fully explore the GUI of a software product. Obviously, this hinders the timeliness of product development and deployment processes.
As another drawback, random execution of the GUI of a software application does not promote a structured, sequential approach to testing, an approach often required to identify the source of bugs. For example, a dialog box displayed by the user interface may consist of various graphics elements including text, a button, and a list box. Activation of the button can result in the display of an entirely new dialog box by the user interface, representing an entirely new state of the application. In order to identify errors that occur as a result of transitioning between states, or to identify the graphics element or actions that correspond to a particular error, the software tester must know the relationship between various states. Unfortunately, this relationship cannot be easily established when executing test monkeys, as actions can be executed in a non-sequential (random) order. Monkey testing, while automatic, is not systematic in its approach to exercising the features of an application.
To overcome the limitations described above, a way is needed to exercise a GUI of an application quickly and systematically, such that the time and cost associated with debugging and testing are minimized. Furthermore, a way is needed to record, or map, the various states of the software as it executes so that relationships between graphics elements and actions of the application can be easily determined.